The Bivariate 2-Poisson Model for IR

Harter’s 2-Poisson model of Information Retrieval is a univariate model of the raw term frequencies, that does not condition the probabilities on document length [2]. A bivariate stochastic model is thus introduced to extend Harter’s 2-Poisson model, by conditioning the term frequencies of the document to the document length. We assume Harter’s hypothesis: the higher the probability f(X = x|L = l) of the term frequency X = x is in a document of length l, the more relevant that document is. The new generalization of the 2-Poisson model has 5 parameters that are learned term by term through the EM algorithm over term frequencies data. We explore the following frameworks: